Read my lips: an animated face helps communicate musical lyrics

Understanding the lyrics of many contemporary songs is difficult. Watching the talker’s face improves speech understanding when the speech is degraded by noise or hearing difficulty. To explore whether the face can be similarly helpful in music, 34 phrases from the song “The Pressman” by Primus (1993) were played to thirteen college students. These phrases were aligned with Baldi , a computer-animated talking head. There were three presentation conditions: original audio, Baldi’s mouthing of the lyrics, and the auditory lyrics aligned with Baldi. The students were asked to watch and listen and to type in as many words as they could understand. Performance was significantly better in the bimodal condition than the auditory condition, showing that visual information from the face contributes to the recognition of musical lyrics. The contribution of the face was somewhat small relative to that found in speech, however, and reasons for this difference remain to be determined.